In a network environment, a network adapter or controller on a host computer, such as an Ethernet controller, Fibre Channel controller, etc., will receive Input/Output (I/O) requests or responses to I/O requests initiated from the host computer. Often, the host computer operating system includes a device driver to communicate with the network controller hardware to manage I/O requests to transmit over a network. The host computer may also utilize a protocol which packages data to be transmitted over the network into packets, each of which contains a destination address as well as a portion of the data to be transmitted. Data packets received at the network controller are often stored in a packet buffer. A transport protocol layer can process the packets received by the network controller that are stored in the packet buffer, and access any I/O commands or data embedded in the packet.
For instance, the computer may employ the TCP/IP (Transmission Control Protocol/Internet Protocol) to encode and address data for transmission, and to decode and access the payload data in the TCP/IP packets received at the network controller. IP specifies the format of packets, also called datagrams, and the addressing scheme. TCP is a higher level protocol which establishes a connection between a destination and a source and provides a byte-stream, reliable, full-duplex transport service. Another protocol, Remote Direct Memory Access (RDMA) on top of TCP provides, among other operations, direct placement of data at a specified memory location at the destination.
A device driver, program or operating system can utilize significant host processor resources to handle network transmission requests to the network controller. One technique to reduce the load on the host processor is the use of a TCP/IP Offload Engine (TOE) in which TCP/IP protocol related operations are carried out in the network controller hardware as opposed to the device driver or other host software, thereby saving the host processor from having to perform some or all of the TCP/IP protocol related operations. Similarly, an RDMA-enabled Network Interface Controller (RNIC) offloads RDMA and transport related operations from the host processor(s).
The operating system of a computer typically utilizes a virtual memory space which is often much larger than the memory space of the physical memory of the computer. FIG. 1 shows an example of a typical system translation and protection table (TPT) 60 which the operating system utilizes to map virtual memory addresses to real physical memory addresses with protection at the process level.
In some known designs, an I/O device such as a network controller or a storage controller may access large amounts of memory to maintain connection and message context information. For example, some network controllers use host or system memory to maintain context or payload information instead of using local memory.
The address of the application buffer which is the destination of the Remote Direct Memory Access (RDMA) operation is frequently carried in the RDMA packets in some form of a buffer identifier and a virtual address or offset. The buffer identifier identifies which buffer the data is to be written to or read from. The virtual address or offset carried by the packets identifies the location within the identified buffer for the specified direct memory operation.
In order to perform direct data placement, an I/O device typically maintains its own translation and protection table, an example of which is shown at 70 in FIG. 2. The device TPT 70 contains data structures 72a, 72b, 72c . . . 72n, each of which is used to control access to a particular buffer as identified by an associated buffer identifier of the buffer identifiers 74a, 74b, 74c . . . 74n. The device TPT 70 further contains data structures 76a, 76b, 76c . . . 76n, each of which is a Translation Entry (TE) used to translate the buffer identifier and virtual address or offset into physical memory addresses of the particular buffer identified by the associated buffer identifier 74a, 74b, 74c . . . 74n. Thus, for example, the TE 76a of the TPT 70 is used by the I/O device to perform address translation for the buffer identified by the identifier 74a. Similarly, the data structure 72a is used by the I/O device to perform protection checks for the buffer identified by the buffer identifier 74a. The TE's 76a, 76b . . . 76n of the TPT 70 may be collectively referred to as an Address Translation Table (ATT) which typically is a portion of the TPT 70. The address translation and protection checks may be performed prior to direct data placement of the payload contained in a packet received from the network or prior to sending the data out on the network.
A buffer identifier and associated virtual address or offset may be used to generate an index to the TE's of the I/O device TPT 70. Each TE typically contains the physical address of a single page.
In order to facilitate high-speed data transfer, a device TPT such as the TPT 70 is typically managed by the I/O device, the driver software for the device or both. A device TPT can occupy a relatively large amount of memory. For example, each TE can occupy 64 bits. As a consequence, a TPT is frequently resident in the system or host memory. The I/O device may maintain a cache of a portion of the device TPT to reduce access delays.